Plenoptic cameras have recently been developed as a method to capture an image of a scene that can be refocused after image capture using appropriate image processing. FIG. 3 illustrates a configuration for a plenoptic camera 200 as described in U.S. Pat. No. 7,936,392 to Ng et al., entitled “Imaging arrangements and methods therefor,” The plenoptic camera 200 includes a microlens array 215 positioned between a main imaging lens 205 and a sensor array 220.
To enable plenoptic imaging, the imaging lens 205 is focused so that the image plane (corresponding to nominal object plane 210) is located at the plane of the microlens array 215. The sensor array 220 is positioned so that each of the individual microlenses in the microlens array 215 forms an image of the aperture of the imaging lens 205 on the sensor array 220. It can be seen that each pixel of the sensor array 220 therefore senses the imaging light falling on the microlens array 215 at a particular position (corresponding to the position of the corresponding microlens) from a particular direction (corresponding to the portion of the imaging lens aperture that is imaged onto that pixel). For example, the imaging light for imaging rays 230, 232 and 234 will be captured by sensor pixels 240, 242 and 244, respectively. The combination of the microlens array 215 and the sensor array 220 can therefore be viewed as a ray sensor 225 that provides information about the intensity about the rays falling on the ray sensor as a function of position and incidence angle.
Ray sensor images captured by the ray sensor 225 can be processed to provide a refocusable imaging mode wherein refocused images corresponding to different focus settings are assembled by combining pixels corresponding to the appropriate imaging rays. This is illustrated in FIGS. 4A-4C.
In FIG. 4A, the desired focus setting corresponds to the original focus setting of the imaging lens 205. In this case, the imaging rays that should be combined to determine the image pixel value for pixel position 250 are shown by ray bundle 252. This corresponds to the trivial case where the rays that would be combined for a particular pixel position are the rays falling on a corresponding microlens in the microlens array 215. It can be seen that the spatial resolution of the refocused image is therefore limited to the spatial resolution of the microlens array 215.
FIG. 4B illustrates the case where a refocused image is determined corresponding to an object plane that is farther away from the plenoptic camera 200 (FIG. 3) than the nominal object plane 210 (FIG. 3). The goal is to determine the image that would have been sensed if an image sensor had been placed at a virtual sensor location 264. In this case, the imaging rays that should be combined for pixel position 250 are shown by ray bundle 254. It can be seen that these imaging rays fall onto the ray sensor 225 at a variety of different spatial positions and angles. The pixel value for the pixel position 250 in the refocused image is determined by combining the pixels in the captured ray sensor image corresponding to the imaging rays in the ray bundle 254.
Similarly, FIG. 4C illustrates the case where a refocused image is determined corresponding to an object plane that is closer to the plenoptic camera 200 (FIG. 3) than the nominal object plane 210 (FIG. 3), having a corresponding virtual sensor location 266. In this case, the imaging rays that should be combined for pixel position 250 are shown by ray bundle 256. In this case, the pixel value for the pixel position 250 in the refocused image is determined by combining the pixels in the captured ray sensor image corresponding to the imaging rays in the ray bundle 256.
With conventional digital camera systems, if a focus error was made during image capture so that the scene object of interest is out of focus, there is no way to correct the focus error post capture. An advantage of the plenoptic imaging system of FIG. 3 is that the focus position of a captured image can be adjusted at a later time after the image has been captured. For example, a user interface can be provided that enables a user to evaluate refocused image corresponding to different focus positions and save the refocused image corresponding to the preferred focus position. However, a disadvantage of plenoptic cameras is that the refocused images necessarily have a substantially lower spatial resolution that the native spatial resolution of the sensor array 220. This reduction in resolution is typically by a factor of 16× to 36×. As a result, the image quality of the refocused image will be significantly lower than a properly focused image captured using a conventional digital camera system using the same sensor array 220.
U.S. Patent Application Publication 2010/0026852 to Ng et al., entitled “Variable imaging arrangements and methods therefor,” provides a method for switching between a low resolution refocusable mode and a higher resolution mode. The method is based on moving the imaging sensor closer to the microlens array. However, even when the imaging sensor is in direct contact with the microlens array, the microlenses will still impart artifacts to the captured image that effectively reduces the resolution of the captured image. For example, the intersection lines between the microlenses will impart repetitive aberrations in the captured image and the thickness of the microlens array will make it impossible to position the sensor at the focus plane of the main lens.
There remains a need for a method to enable a camera system to be switched or changed between a low resolution refocusable mode and a high resolution non-refocusable mode.